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ABSTRACT 



This discussion of strategies for improving writing 



evaluation in a course in English as a Second Language begins with a 
brief review of relevant research and then offers concrete 
suggestions for improvement in testing. The suggestions are in the 
form of five general considerations rather than specific test-writing 
techniques. They include the following: (1) provide plenty of 
opportunity for students at all levels to practice the type of 
writing expected on the test; (2) make the assessment reflect the 
goal of the course, with mechanical and grammatical errors kept in 
the perspective of the overall communicative purpose of the text; (3) 
test a variety of skills and create writing tasks of varying lengths; 
(4) plan tests to be shorter and more frequent; and (5) create 
evaluation checklists or profiles to systematize grading, reduce 
bias, and communicate results to students. Ways in which curriculum 
and classroom activities can be adjusted to reflect these 
considerations and better prepare students for evaluation are 
offered. A 15-item bibliography is included. Sample grading criteria 
for compositions and sample criteria for use in a composition course 
(presented in Spanish) are appended. (MSE) 
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Formulation of Tasks and Evaluation —-^^-^^uAiiQA 
David L. Paulson 
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University of Illinois at Urbdna-Champaign center rERic).- 

W Writing. The "other" productive language ability. The step-child 

^ of the foreign language curriculum. Like the old bachelor uncle or 

^ maiden aunt in the family, its position in the curriculum is secure but 

Q 

W its role is not of ten well defined. In grammar-translation methodology, 
the role of writing was to produce accurate translations . In the audio- 
lingual framework, writing was the last of the four language skills to 
be addressed and was judged for its reproduction of the model response. 
In communicative langauge teaching methods we look at expressing ideas . 
It is in that framework that this paper shall address assessing writing 
ability and the issues involved in its measurement. 

Rather than waiting for the backwash from the changes in testing 
writing at a national level and also from testing ESL writing, it is 
incumbent upon the second language acquisition researcher to identify 
where testing of foreign language writing currently stands, to identify 
those broad issues in testing that are relevant to testing foreign 
language writing and to carefully reexamine the entire testing process 
from conceptualization of writing goals to the development of the 
testing instrument. Testing is receiving renewed national attention. 
The scope of the issue of testing is not limited to just the classroom. 
There are highly-charged ideological, methodological, political and 
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technical problems to contend with in taking a new, critical look at 
national school testing (Merl, 1991) . While the focus of this paper is 
not on national school testing, I mention this to underscore the 
imriiediacy of the issue of testing writing in a foreign language 
curriculum. 

Omaggio (1986) indicated that many classroom a -hievement tests 
still reflect the thinking and philosophies of the sixties and now out- 
moded behaviorist psychology in that many tests are still discrete 
point. Instead of reflecting philosophies of testing, tests should 
reflect course goals. Valette (1978) observed that the tests and 
grading methods used in the classroom are the ultimate guide for 
identifying instructional objectives (or lack thereof) . Our tests are 
the true measure of what is important to us; they may reveal the short- 
sightedness of our goals and the limitations of our imagination in 
measuring them. 

Hairston (1986) indicates that there has been a shift towards 
focussing on the process of writing rather than the written product. She 
indicates that the term "the process of writing" does not refer to a 
singular process; instead, "process" refers to a constellation of 
processes and techniques. Flower & Hayes (1981) state that writing 
involves a process of making plans and carrying them out; it can be 
analyzed and mapped. Hughey ec . (1983) encourage a great deal of list 
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making and map making throughout the process of writing. 

Hughey et al. (1983) first state that the teacher's role is to 
understand the "additional complexities involved in learning the 
writing process. . . " (p. 47, emphasis added) . They then go on to identify 
three ways the teacher can create a "context for learning" and 
consequently, a framework for testing: 1) select interesting and 
stimulating material for testing; 2) do not assign use of a rhetorical 
structure but rather let that structure emerge as the student becomes 
aware of the options through the organizational process; 3) be prepared 
to deal with syntactic structures that the rhetorical structure will 
generate (p. 47). Hughey etal. (1983) then go on to specif y a long list 
of instructional objectives and responsibilities for the role of the 
teacher. The teacher is characterized as imparter of specific 
knowledge, provider of necessary organizational skills, simulacrum of 
the "real audience", and evaluator of the product. 

But how "real" is the teacher as the sole member of the audience? 
The results of following even the most effective process of writing 
remain in a vacuum of sorts when the only plausible audience is the 
teacher. Those of us with experience as composition instructors know 
that telling the class to "write a paragraph to the president of this 
university explaining why there should be more Spanish classes offered 
each semester" is still just an artificial paragraph to the instructor 
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unless, of course, they are actually sent to the president. There is no 
question as to the importance of establishing a real audience in 
writing. Even a personal journal has an audience - the writer. 

By establishing a frame or context within which the writing will 
be used to communicate may contribute greatly to the face validity of 
the testing instrument (See Savignon 1983) . With the absence of a 
specified implied reader in a testing situation, where the learner is 
even more keenly aware of the real reader, the creation of specific 
writing purposes and roles that come bundled with context and meaning 
may well serve to help the learner better focus in on and consequently 
more clearly create the written message. 

Rather than pursuing specific ways to test /friting, based on the 
preceding observations I shall instead present the following general 
considerations for improving the testing of writing: 

1) Provide plenty of opportunities for students at all levels to 
practice the type of writing that you expect them to do on the writing 
test. 

Often, writing is avoided by instructors and students alike and 
saved for exam time. For example, in a second semester course where 
content may begin to shift towards social issues and other extra- 
curricular discussion topics, giving the students an essay test on an 
issue can be disastrous if they have only had the opportunity to listen 
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to and possibly discuss the issues with no opportunity to write about 
them. Periodic quizzes can be administered to measure student 
comprehension of the broad issues and necessary vocabulary before 
beginning to write about differing opinions and specific issues. Or, 
for example, in an introductory language classroom setting, students 
can practice short expository passages (e.g., writing descriptions of 
each other or of objects around them) before a testing situation where 
they are expected to write a description in order to use specific 
vocabulary. 

Instructors who struggled for years to learn a foreign language as 
well as those who are native speakers often lose sight of the fact that 
learners have an incomplete second language linguistic system. It may 
not be realistic to formulate a writing task in a testing context that 
measurer: orthography, syntax, accent marks and vocabulary, and then 
to expect an answer that conveys correct information. For many 
learners, this is cognitive overload. This first proposal essentially 
addresses the formulation of the writing task in global terms. 
Proposals 2 through 4 address these specific issues in the formulation 
of writing tasks . 

2) Assessment of writing should reflect the goal of the course - 
errors in accentuation, spelling, and syntax must be kept in 
perspective with the overall communicative purpose of the test. 
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"Make sure all your accent marks are where they belong, " says the 
teacher, "I' 11 be taking off half a point for each one that' s missing - 
and that includes accents that are floating around above the word, too ! " 
Don' t laugh - many of us have said this to our students with the best of 
intentions . Realistically, students begin to have only a modest control 
of diacritics in writing at the sentence level after several semesters 
of study. 

It has been demonstrated (Krashen 1984, Semke 1984, Zamel 1985) 
that the teacher's written corrections, often referred to as "the red 
pen", can do far more harm than good in helping students to produce 
written language that is more coherent (i.e., sticks to the subject at 
hand and makes sense) and mechanically precise (e.g., accent marks, 
spelling, syntax, etc.). Semke (1984) reported that marking formal 
errors on students' free-writing assignments was ineffective for 
increasing writing accuracy, and went on to suggest that the effect 
could even be detrimental since students reported that they had a more 
negative attitude toward the writing experience as a result of the 
teacher' s written corrections. 

Is this to say that we should let errors in diacritics, spelling, 
grammar, etc., just slide? Not necessarily. We must keep the issue of 
accent mistakes and omissions in perspective in each particular test. 
In other words, if the purpose of the writing test is to determine if 
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students know where accents go or how to spell vosotros endings (2nd 
person plural familiar pronoun in Spanish) , then make that purpose 
explicit and count off as much as seems appropriate for missed accents 
and misspellings on those specific items and/or devise editing 
exercises that help them master those tasks. 

Complete omission of diacritics in written Spanish does not make 
it incomprehensible. However, omitting diacritics does create 
ambiguity at times . This also may make a more global statement about the 
second language literacy of the writer. When a student asks, "How much 
do accents count on the exam?", a more appropriate response could be, 
"accent marks contribute to the meaning - without them the meaning may 
be unclear. The purpose of this test is for you to write as clearly and 
concisely as you can in Spanish. " Then, when che student brings up her 
paper after the exam, rather than asking, "Did you check that all your 
accent marks are correct?", the teacher can ask, "Did you read this back 
to yourself? Is it clear? Does it say what YOU want to say?" Having the 
student say what SHE wants to say brings our philosophy of promoting 
communicative competence in the classroom (See Savignon 1983) into the 
reality of testing. This approach is also useful in helping the teacher 
"save face" when students discover that many native speakers' use of 
accents is inaccurate, inconsistent or non-existent. 

3) It is important to test a variety of writing skills and to create 



8 



I 



8 

tasks of varying lengths. 

It is not uncommon to hear students in the intermediate-level 
composition course lament that they never had to write anything longer 
than a one-sentence response to a specific question throughout their 
basic language instruction. And, to further sterilize their prior 
writing experiences, their written response was most often graded on 
its grammatical accuracy - rarely on its content. Traditional workbook 
exercises provide students with opportunities to write in the foreign 
language from the beginning of most curricula. Traditional workbooks 
are often the only writing practice students have, which can give the 
impression that writing is something they do because they have to, that 
it should be done alone, at home, when no one else is around, and that 
it' s mechanical and dull . 

Grading workbook exercises is also very tedious. Nevertheless, 
many teachers make tests that are based largely on these workbook 
exercises. I do not propose that we discontinue using traditional 
workbooks . I do propose that we keep in mind that workbooks are usually 
primarily for grammar and vocabulary practice. Writing solely in 
workbooks may be undermining the entire notion of writing as 
communication. Students needmore non-workbookwritingexperiences and 
non-workbook style writing tests to offer them a broader base of writing 
skills . 
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Many supplemental writing activities can be introduced into an 
introductory course that students can use to practice diverse registers 
of writing. Forexample, filling out forms, writing letters, responding 
to a quastionnaire, writing descriptions and directions, etc. (See 
Hedge 1.988 for numerous writing activities) . These varied writing 
activities all have ecological validity as they represent vital, real- 
world, and completely different kinds of writing skills. The most 
dynamic writing activity I have ever directed was having a group of 
migrant workers, to whom I was teaching basic literacy skills and ESL, 
write to the White House, asking that the funding for their classes not 
be cut. If their classes were cut, they would have no other formal 
instruction in English and would risk not having their legal working 
papers renewed. These men knew that their letters could make a 
difference and that there was much at risk. 

We do not often have the same o^^portunity for that degree of 
immediacy in writing tasks in a foreign language classroom, but 
certainly we can provide more opportunities for writing that impact the 
students' lives in ways other than just a grade. 

4) Plan tests to be shorter and more frequent. 

Greenia (in press) describes a framework for a writing course that 
offers frequent, varied activities rather than a few massive writing 
activities that terrify students and are a nightmare to grade. This 
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seems more realistic and productive . Greenia goes on to show that lower 
division courses can follow the same principle of frequency and variety 
in formulating writing tasks. Just as a composition course need not 
culminate in a research paper, nor does a lower-level language course 
need to have a final writing activity that will pull all these varied 
experiences together. It may be enough for these varied experiences to 
stand as they are without a final cumulative written project. The fifth 
proposal examines writing checklists and profiles as flexible and 
consistent means of evaluating writing. 

5) Writing evaluation checklists or profiles help protect the 
student against a biased or less-experieiiced teacher/evaluator and help 
the teacher explain her reasons for the evaluation (See Appendix for 
examples) . 

Once a variety of writing tasks have been formulated, a flexible 
means of evaluating those tasks must be designed. These evaluation 
profiles also tend to systematize an otherwise nebulous and frustrating 
grading procedure and make it go more quickly. IT you are testing 
writing as defined as prose that conveys mf:aning, then grade the 
writing. Too often writing is synonymous with grammar, and whenever a 
student writes, the grammatical accuracy is first and most important. 
When evaluating longer answers or essays, checklists or other holistic 
evaluation criteria are tools that offer teachers a means of looking at 
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writing from a more "objective" standpoint. The evaluation forms guide 
teachers through specific questions about the students' essays, often 
in terms of, for example, content, comprehensibility, vocabulary, 
grammar, and style. Different evaluation checklists give different 
weights to these specified areas of evaluation. There is no single 
correct formula. The teacher must simply choose, or better yet, create 
a checklist that best reflects the goals of the course itself . A grammar 
course with a writing component could give more weight to grammar, 
whereas an introductory language course could give more weight to 
content, and a composition course more weight to organization. Students 
can use evaluation checklists as well to evaluate each other' s writing . 

Profiles and checklists help steer teachers away from just looking 
at one aspect of essays or from being biased against or for particular 
essays due to structure or content. Grading a student' s writing with an 
evaluation checklist gives the student more questions to ask afterwards 
regarding her composition, and gives the teacher more help in answering 
student questions about how she arrived at the particular grade. When 
the evaluation profile is given to the student in advance, the checklist 
helps her remember to focus on all important aspects of writing instead 
of just grammar. 

I taught more than 400 students EFL composition at one time at the 
University of Barcelona during the 89-90 academic year. I found that 
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having an evaluation profile saved my life when I simply could not have 
reraembered why I had given a grade on a particular composition if the 
stuQv=nt had a question. In Spanish composition classes at the 
University of Illinois, a different profile is used for every essay 
assignment since different styles and techniques are stressed in each 
module of the course. This may seem cumbersome, yet testing is also 
supposed to help the teacher evaluate whether the students learned what 
was taught . A very specific profile helps to measure this with far more 
accuracy. 

Responding to student writing with the use of an evaluation 
profile directs the teacher's comments to very specific areas of 
content and style of the student writing. This directed and specific 
feedback to the students opens doors for much more interaction between 
the student and the teacher, as well as between students. Detailed 
explanation of the parts of an evaluation profile and syntheses of 
results of evaluations of student writing can serve as points of 
departure for future lesson plans, creating a more context-rich rather 
than structure-rigid classroom environment. Instruction of specific 
grammar points and accentuation problems that have arisen or are 
anticipated can then be incorporated into a lesson as part of a holistic 
approach to teaching writing at the beginning level all the way through 
to advanced courses which require highly technical skills of 
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integrative genre such as literary analysis. 

Taylor (1991) acknowledged the helpful role that these processes 
for organizing writing can have, but went on to underscore the 
importance of establishing the classroom as a discourse community by 
means of having students regularly share their writing with one 
another . In the third-year Spanish Composition course at the University 
of Illinois at Urbana-Champaign, students use electronic mail weekly 
to write short reactions in Spanish to topics of current interest posted 
either by the instructors or other students in the course. They also 
keep dialog journals which they exchange with a "journal partner" every 
week. These are effective ways to stimulate interest in writing. Within 
this discoursal framework, diverse skills can be practiced and then be 
more fairly tested. 

An earlier version of this paper was originally presented at the 1991 
Annual Meeting of American Association of Teachers of Spanish and 
Portuguese in Chicago, Illinois. 

The author wishes to thank Jane Berne, Donna Binkowski, James F. Lee and 
Joyce Tolliver (Univer si ty of Illinois at Urbana-Champaign) , and George 
Greenia (College of William and Mary) for their invaluable comments on 
earlier drafts of this paper. 
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APPENDIX 

SAMPLE GRADING CRITERIA FOR COMPOSITIONS 



STANDARD CRITERIA FOR BEGINNING LANGUAGE COURSES 



COMPREHENSIBILITY 



Most is incomprehensible 
Generally comprehensible, but 
needs interpretation 

Completely comprehensible 



COHESIVENESS 



Composition is a series of 

separate sentences with no 

transitions 
Composition is choppy or 

disjointed 
Composition flows smoothly 

and has some style 



INFO CONVEYED 



Minimal information given 



Info adequate for topic 



VOCABULARY 



Very complete info given 

Inadequate /repetitious/ 
inaccurate/erroneous 

Adequate, but contains many 
errors 

Quite broad in range, 
precise and expressive 



TOTAL : 
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SAMPLE CRITERIA FOR A COMPOSITION COURSE 

Composici6n 1: La exposici6n 
Puntos centrales: 

I. La validez y la claridad de la idea central ("main point") del 
ensayo 

II. El enfoque; la delirriitaci6n del tema 
Criterios : 

I. A. iTiene el trabajo una idea central v^lida, expresada 
claramente en una o dos frases? 

Si : seguir a B. 

No: Hay que establecer antes de proceder cu^l es la idea central 
del ensayo . 

B. ^Esta expresada explicitmente esta idea central al final de 
la introducci6n o secci6n preliminar? 

Si : Seguir a E . 
No: Seguir a C. 

C. ^Esta expresada explicitamente esta idea central al final 
del ensayo? 

Si : Seguir a D. 

No: 0 el trabajo no tiene idea central, o esta metida al medio del 
ensayo . En cualquier de los casos, hay que expresar una idea 
central en el lugar apropiado. 

D. Si el punto central est^ expresada explicitamente al final 
del ensayo, tambi^n tiene el ensayo puesto, al final de la 
introducci6n, un "punto preliminar" que capte el interns del 
lector y que le impulse a seguir leyendo? 

Si : Seguir a E. 

No: Hay que revisar el final de la introducci6n para que exprese 
un "punto preliminar." Este "punto preliminar" puede 
plantear una pregunta, expresar una meta, hacer una 
observaci6n clave, o en alguna otra forma anticipada para el 
lector las ideas que van a seguir. 
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E. lEs v^lida y relevante la idea central del ensayo? Nota : Para 
determinar la validez de la idea central^ hay que determinar 
para qui6n est^ escrito el ensayo (qui6n es el lector 
anticipado) y cu^l es el prop6sito del ensayo (que intenta el 
autor lograr con la escritura del ensayo) . La idea central 
es v^lida y relevante si cumple bien el pr6posito^ y si toma 
bien en cuenta las necesidades del lector anticipado. 

Si: Seguir a parte II. 

No: El ensayo necesita una nueva idea central. 

II. ^Desarrolla bien el ensayo los temas y conceptos mencionados al 
final de la introducci6n? 

A. Generalmente aparecen 2-3 veces por p^gina las "palabras 
tem^ticas" mencionadas al final de la introducci6n? 

Si : Seguir a B. 

No : Hay que revisar el ensayo para poder mencionar^ de una manera 
natural, las "palabras tem^ticas" m^s f recuentemente. 

B. ^Tienen una relaci6n natural y 16gica las ideas expresadas 
en cada p^rraf o con la idea central, tal como la expresan las 
"palabras tem^ticas" y las oraciones temciticas? 

Si: Ha cumplido con los criterios. 

No: Hay que revisar el ensayo para que las ideas expresadas se 
vinculen m^s estrechamente con la idea central del ensayo. 

Evaluaci6n : 

Puntos centrales (versi6n 1) : x . 90 = / 90% 

Forma {gram^tica, vocabulario, estilo} (versi6n 2) : 

X. 10 = / 10% 

NOTA TOTAL (versi6n 3)= % 
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Gran parte de la elaboracion de los criterios para esta composicion se 
debe al Prof. Gregory Colomb (Business & Technical Writing), "A 
checklist f ormarking/revising termpapers andreports, " ms . University 
of Illinois at Urbana-Champaign . 



